1. Field of Invention
The present patent document is directed towards systems and methods for data processing. More particularly, the present patent document is directed towards systems and methods that facilitate the handling of information between an in-memory database system and a data warehouse/analytics system.
2. Description of the Related Art
As more people and businesses use computers and as more services are provided electronically, today's modern applications are being required to receive, processes, and store vastly larger amounts of data. Furthermore, today's modern applications are also being required to respond faster and capture more information to allow businesses to perform the analyses needed for the making the best business decisions. Problems arise, however, because different processes have inherent weaknesses. When these systems interact, these weaknesses can compound, resulting in a system that, in some ways, is less than the sum of its parts.
Consider, for example, the current implementation of online transactional processing (OLTP) products and online analytical processing (OLAP) products. FIG. 1 depicts an in-memory database 105, which is an OLTP product, communicably coupled to a data warehouse 110, which is an OLAP product. OLTP products are systems that usually deal with a large number of real-time transaction-related applications. For example, as shown in FIG. 1, the in-memory database system 105 receives a plurality of real-time transactions 150. OLTP products' relative simplicity and efficiency allow them to be very responsive to these interactions—thereby allowing them to support near real-time data analysis and decision making. Hence, such products are referred to as “Fast Data.” However, these “Fast Data” systems tend to operate on limited amounts of data that may be stored in its in-memory database. Huge volumes of data present significant challenges to OLTP products. Limited by the in-memory database(s), they cannot handle enough data needed for deep analytics.
Online analytical processing (OLAP) products, such as data warehouse products, are able to store and analyze much larger quantities of data. The vast data storage allows for more complex, deep, and spanning analytics to be performed; and hence, such products are sometimes referred to as “Big Data.” However, like all things that are big, they tend to be slower. OLAP systems are slower for at least two reasons. First, because OLAP products receive and process significant amounts of data, they tend to receive the data in large batches, such as overnight batch jobs. For example, the in-memory database 105 might receive many data transactions 150 during the course of a certain time period, like a single day. The in-memory database 150 stores these transactions in its database for that day, and then during the evening, it 105 transmits the data in a batch 155 to the data warehouse 110. Because the data warehouse only receives updates at certain intervals, there is inherently a delay in its ability to act upon that data. Put simply, delays in receiving the data means delays in processing the data. And second, the structure and components of OLAP products tend to operate slower than in-memory systems.
While the currently-used implementation depict in FIG. 1 is useful, its “Big Data/Fast Data” duality has significant drawbacks. This two system approach results in data being stored in different systems, spanning different times, and with different abilities to analyze the data. Thus, a significant problem exists if someone wants to, in real-time (or near real-time) analyze newly received data with historical data. The current implementations cannot meet these demands due to limited query performance, limited amounts of data, and inaccessibility of the data.
Accordingly, what is need are systems and methods that can overcome the limitations of the bifurcated Big Data/Fast Data system to produce better, more comprehensive, and faster data analyses.